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EVALUATION  OF  THE  A-7  REQUIREMENTS  DOCUMENT 
BY  ANALYSIS  OF  CHANGE  DATA 


INTRODUCTION 

In  recent  years  a  number  of  techniques  for  improving  the  reliability  and  decreasing  the  cost  of 
software  have  been  suggested.  These  techniques  deal  with  various  aspects  of  the  software  life  cycle. 
An  integrated  set  of  two  or  more  techniques  covering  one  or  more  phases  of  the  life  cycle  may  be 
defined  as  a  methodology.  It  is  not  obvious  how  to  refine  and  adjust  the  basic  techniques  in  a  metho¬ 
dology  for  the  individual  factors  of  some  specific  environment  and  application.  Software  engineering 
involves  the  application  of  a  methodology  to  a  particular  environment. 

The  software  community  is  interested  in  the  analysis  of  techniques,  their  integration  into  a 
methodology,  and  the  engineering  of  that  methodology  to  particular  environments.  An  effective  way  to 
evaluate  a  methodology,  understand  the  environment,  and  refine  the  methodology  for  the  environment 
is  to  collect  data  that  characterize  the  methodology  and  the  environment  and  supply  insight  into  both. 

A  major  source  of  insight  when  analyzing  a  software  development  project  is  a  record  of  the 
changes,  including  error  corrections,  made  as  the  development  progresses.  Data  showing  where 
changes  were  made,  what  kinds  of  changes  were  made,  and  the  effort  involved  in  making  changes  can 
be  used  to  evaluate  methodologies,  characterize  environments,  and  permit  the  proper  engineering  of 
the  methodologies  for  the  environments. 

We  describe  in  this  report  an  effective  data  collection  method,  from  definition  of  objectives  of  the 
data  collection  to  analysis  of  results.  We  show  how  analysis  of  the  data  can  answer  questions  with 
respect  to  how  successfully  the  goals  of  the  development  methodology  are  met.  The  A-7  requirements 
document  is  used  as  an  example.  We  provide  the  results  of  data  analyses  conducted  partway  through 
the  A-7  flight  software  development  cycle,  and  we  discuss  the  utility  of  information  obtained  by  such 
partial  analyses.  The  next  section  describes  the  project  studied  and  its  overall  objectives.  The  third  sec¬ 
tion  discusses  the  relation  between  the  data  collection  methodology  and  the  software  development 
methodology.  The  fourth  section  presents  the  data  and  its  analysis.  The  fifth  section  presents  the  con¬ 
clusions  and  some  suggestions  for  further  studies. 

A-7  PROJECT  OVERVIEW 

A  significant  obstacle  in  the  field  of  software  engineering  is  lack  of  technology  tiansfer.  Many 
techniques  are  developed  in  academic  environments  or  in  the  construction  of  small  programs,  Large- 
scale  software  developers  are  reluctant  to  use  techniques  that  have  not  been  tested  in  the  development 
of  complex  systems.  The  Naval  Research  Laboratory  (NRL)  and  the  Naval  Weapons  Center  (NWC't 
are  redesigning  and  rebuilding  the  operational  flight  program  for  the  A-7  aircraft  using  techniques  such 
as  information  hiding  [1,2],  formal  specifications  [3-6],  abstract  interfaces  [7],  cooperating  sequential 
processes  [8,91,  process  synchronization  routines  [81,  and  resource  monitors  [10-12).  The  goals  are  to 
demonstrate  the  feasibility  of  using  these  techniques  to  develop  a  complex,  real-time  program  and  to 
provide  the  Navy  with  a  model  for  the  development  of  avionics  programs.  The  techniques  to  be  used 
were  selected  because  they  are  claimed  to  facilitate  the  development  of  software  that  is  reliable,  easy  to 
change,  easy  to  understand,  and  easy  to  demonstrate  correct. 
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The  characteristics  ol'  the  A-7  operational  flight  program  and  the  constraints  on  the  redevelopment 
project  are  described  in  Ref.  13.  The  requirements  description  was  completed  in  November  1978.  and 
the  program  is  currently  in  the  design  stage. 


DATA  COLLECTION 
Goals 


The  opportunity  to  apply  recent  software  engineering  technology  in  the  development  of  a  complex 
model  system  does  not  seem  to  occur  often.  We  considered  it  important  not  to  lose  the  chance  to 
monitor  closely  the  progress  of  the  project.  A  separate  data  collection  effort  to  permit  evaluation  of  the 
project  during  the  development  cycle  was  established.  Final  evaluation  of  the  success  of  the  rebuilt  A-7 
software  must  await  the  delivery  and  use  of  the  software.  A  number  of  intermediate  evaluation  points 
may  be  established  to  provide  some  insight  into  the  redevelopment  process.  The  intermediate  evalua¬ 
tions  may  be  based  on  the  goals  established  at  each  phase  of  the  project  and  on  the  goals  established  for 
the  different  techniques  used.  As  an  example,  Heninger  has  described  the  following  six  objectives  of 
the  requirements  document  [13,14]: 


•  Specify  external  behavior  only. 

•  Specify  constraints  on  the  implementation. 

•  Be  easy  to  change. 

•  Serve  as  a  reference  tool. 

•  Record  forethought  about  the  life  cycle  of  the  system. 

•  Characterize  acceptable  responses  to  undesired  events. 

The  main  purpose  of  the  data  collection  and  analysis  described  here  is  to  help  measure  the  success 
with  which  the  preceding  objectives  are  met. 

Identification  of  Data  to  be  Collected 

Once  the  decision  to  monitor  the  project  was  made  and  the  objectives  for  the  document  were 
clearly  stated,  the  next  step  was  to  identify  the  data  to  be  collected.  To  do  this  we  established  a  list  of 
questions,  the  answer  to  each  question  helping  to  measure  the  success  of  attainment  of  an  objective. 
As  an  example,  consideration  of  the  objective  "Be  easy  to  change"  led  to  the  following  questions: 

•  Is  the  document  easy  to  change? 

•  Is  it  clear  where  a  change  has  to  be  made? 

•  Are  changes  confined  to  a  single  section? 

To  answer  these  questions  we  needed  to  know,  for  each  change,  the  effort  required  to  make  the 
change,  some  measure  of  how  much  of  the  document  had  to  be  examined  to  make  the  change,  and 
how  many  sections  of  the  document  were  actually  modified  when  the  change  was  made.  We  decided  to 
measure  effort  in  man-hours  and  both  the  amount  of  the  document  examined  and  the  amount  modified 
in  making  the  change  in  sections  of  the  document.  The  complete  list  of  questions,  taken  from  Ref.  15, 
is  shown  in  Table  I. 
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Table  1  —  Questions  Used  in  Designing  the  Change  Report  Form 

(1)  Is  externally  visible  behavior  only  specified  without  implying  a  particular  implementation? 

(2)  Are  the  appropriate  external  interfaces  specified? 

(3)  Are  the  external  interfaces  specified  correctly? 

(4)  Is  the  document  easy  to  change? 

(5)  Is  it  clear  where  a  change  has  to  be  made? 

(6)  Are  the  changes  likely  to  occur  predicted  correctly? 

(7)  Are  changes  confined  to  a  single  section? 

(8)  Is  the  proper  set  of  undesired  events  described? 

(9)  Is  the  notation  used  unambiguous? 

(10)  Which  sections  have  the  most  errors? 

(11)  Where  do  the  most  changes  have  to  be  made  ? 

(12)  Which  type  of  tables  has  the  most  errors? 

(13)  Does  the  document  contain  unnecessary  information? 

( 14)  What  use  of  the  document  reveals  the  most  errors? 

(15)  Are  sections  3  (Modes)  and  4  (Functions)  consistent  with  each  other? 

(16)  Is  the  dictionary  complete,  correct,  and  consistent  with  the  rest  of  the  document,  and  will  it 
remain  so? 

(17)  Which  subsections  of  sections  2  (Data  Items),  3  (Modes),  and  4  (Functions)  are  most  error- 
prone  ? 

Forms  Design 

Experience  gained  in  designing  and  using  change  report  forms  for  NASA's  Software  Engineering 
Laboratory  116]  and  for  the  Architecture  Research  Facility  study  117]  helped  considerably  in  the  design 
of  the  A-7  change  report  forms.  Among  the  lessons  learned  from  those  projects  were  the  following: 

•  The  form  should  fit  on  one  piece  of  paper. 

•  Providing  space  on  the  form  for  brief  written  descriptions  of  changes  was  helpful  for  valida¬ 

tion  and  analysis  purposes. 

•  Those  people  who  are  going  to  fill  out  the  forms  should  have  a  chance  to  help  design  them. 

•  Checklists  are  convenient  for  both  collecting  and  analyzing  data. 
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A  prototype  form  was  designed  to  collect  the  data  needed  to  answer  the  questions  described  in  the 
preceding  section  and  circulated  to  all  members  of  the  A-7  project  [15].  The  form  was  modified  and 
the  process  repeated  until  all  were  satisfied  with  the  proposed  form.  It  was  then  briefly  tested,  and  a 
few  minor  modifications  made. 

Data  Collection  Procedures 

Change  data  collection  was  made  part  of  the  configuration  management  process  for  A-7  docu¬ 
ments.  As  documents  are  completed,  they  are  placed  under  configuration  control,  and  all  changes 
made  to  them  are  described  and  monitored.  Change  report  forms  tailored  to  the  objectives  and  format 
of  the  documents  under  control  are  used.  Figure  1  is  the  change  report  form  used  for  the  requirements 
document. 

Proposed  changes  to  baselined  documents  are  submitted  on  a  change  report  form  (CRF).  The 
proposed  change  is  reviewed  by  the  configuration  control  board  (CCB).  If  disapproved,  the  change  may 
be  returned  to  the  proposer  with  an  explanation.  If  approved,  an  A-7  project  member  is  assigned  to 
implement  the  change.  The  implemented  change  is  reviewed  again  by  the  CCB  for  correctness.  Often, 
the  proposer  is  a  member  of  the  CCB.  Also,  the  proposer  is  sometimes  the  same  as  the  implementer. 

Integration  of  change  data  collection  with  configuration  control  has  the  advantage  that  no  change 
data  is  lost  as  long  as  the  configuration  control  process  works.  Furthermore,  only  one  form  is  needed 
for  both  configuration  control  and  data  collection.  Change-data  analysts  are  thus  assured  of  the  com¬ 
pleteness  of  their  data.  In  addition,  the  proposer  and  implementer  of  the  change  are  both  identifiable  if 
further  information  is  needed. 

A  characteristic  of  the  change  process  is  that  trails  to  and  from  the  document  and  the  CRF  are 
maintained.  Changed  sections  are  marked  both  with  a  symbol  to  denote  that  a  change  has  been  made 
and  with  the  number  of  the  CRF  describing  the  change.  The  CRF  always  contains  the  (sub)section(s) 
changed,  and  often  the  page  numbers.  The  change  data  analyst  can  easily  find  the  exact  part(s)  of  the 
document  changed. 

Data  Validation  and  Analysis 

Several  times  a  year  the  accumulated  change  report  forms  are  reviewed  and  an  analysis  conducted 
for  evaluation  purposes.  As  part  of  this  process,  the  forms  are  validated.  Experience  with  previous 
change  data  collection  projects  has  convinced  us  that  validation  of  the  forms  is  essential.  Validation 
includes  examination  of  each  form  for  completeness  and  consistency.  When  necessary,  interviews  with 
the  proposer  and  the  implementer  of  the  change  are  conducted  to  obtain  missing  data  and  correct 
errors. 

The  various  kinds  of  cross-referencing  used  facilitate  both  change  to  the  documentation  and 
change  data  validation  and  analysis  of  the  kind  described  in  this  paper.  As  an  example,  during  change 
validation  several  incompletely  implemented  changes  have  been  discovered  and  reported  back  to  the 
configuration  control  board. 

This  report  contains  the  results  of  the  first  major  evaluation  of  the  changes  to  the  requirements 
document.  These  changes  cover  the  period  from  19  December  1978  to  15  February  1980  (the  docu¬ 
ment  was  baselined  in  November  1978). 

RESULTS  OF  THE  DATA  ANALYSIS 

The  answers  to  the  questions  posed  in  the  preceding  section  are  presented  here,  based  on  data 
collected  during  the  first  14  months  of  use  of  the  requirements  document. 
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A-7  Requ •'  rements  Document  Change  Report  Form 


Number 

Month  Day 

Year 

Current  Date . 

Change  Started  On.. 

a 

Change  D>'  scovery 


1.  How  was  the  document  hein.g  used  when  the  need  for  the  change  was  discovered? 

[3J  Va! idat ion  review  by  the  authors  |33  As  a  software  design  reference 

[3]  Val  idation  review  by  non-authors  QJ  As  a  coding  reference 

|  |  As  a  maintenance  reference  r~1  As  a  test  reference 

| — |  Other: _ 

Identification 

2.  Description  of  change: _  _ 


Section(s) 

Changed 

Sect;cr,(s^  Eyenuned  But  Not  Changed 

Subsection(s) 

Intro. 

0. 

, 

TC  2 

1. 

HH"  -jH 

Data  Items 

2. 

Modes 

3. 

Functions 

4. 

J1 

Timing 

5. 

QESZEUB  ii'i'i 

6. 

|UE's 

7. 

8. 

9. 

DM 

am 

ID  eta  I  ten  Index 

1 

Function  Index 

Dictionary 

Type  Of  Change 

4.  Why  is  the  change  being  made  (check  one)? 

n To  correct  an  original  error 

|  J  To  complete  or  correct  a  previous  change  (Previous  CRF  $  _ ) 

[33  To  comply  with  unexpected  requirement  change  (violates  sect.  9  assumption) 
[33  To  comply  with  expected  requirement  change  (assumed  in  sect.  9  subsect.  ) 

[  J To  remove  unnecessary  information  ^ 

[33  To  reorganize  within  one  section  l  Readability 

[33  To  reorganize  among  several  sections  J  Improvement 

| — l  Other: _ 

5.  What  was  the  effort  in  person-time  required  to  understand  and  maze  the  change? 

# . i . » . ! . i . i 

0  1  raan-hr  1  man-day  1  man-week  1  man-month 

6.  Estimate  the  percentage  of  the  effort  just  to  understand  the  change. 

i . i . ; . • . . ! . ! 

0  20  40  60  80  100 

Fig.  la  —  A-7  requirements  document  change  report  form  (obverse) 
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FOR  ERROR  CORRECTION’S  ONLY 


Type  Of  Error 


7.  How  is  the  error  best  characterized  (check  one)? 

□  Clerical  a  Incorrect  Fact 

[ — |  Ambiguity  Information  put  in  wrong  section  A  Inappropriate 

j — |  Omission  £]  Implementation  fact  included  /  Fact 

Q  Inconsistency  ^30ther:, _ 

For  Section  2  (Data  Itew)  Errors 

8.  Which  part7s7  of  the  subsection  were  incorrect  (check  all  that  apply)? 

|  |  Hardware  Instruction  Sequence 

j — |  Description  Data  Representation 

|  |  Value  Characteristics  □  Comment 

For  Section  3  (Modes)  and  Section  4  (Functions)  Errors 

How  is  the  error  best  characterized  (check  all  that  apply)? 

9.  Section  3  10.  Section  4 


f  |  Mode  Transition  Error 

|  |  Mode  Condition  Error 

f  |  Mode  Overview  Error  A 

|  |  Corresponding  errors  were  made  in 
both  sections  3  and  4. 


|-1  Applicable  Mode  List  Error 
Q  Output  Data  Item  Error 
[ — |  Output  Description  Error 
I — (  Event  Table  Error 
a  Selector  Table  Error 
|  |  Condition  Table  Error 
|  f  Inconsistent  or  incomplete  table 


For  Errors  Involving  Dictionary  Items 
12.  How  is  the  error  best  characterized  (check  all  that  apply)? 
□  Incorrect  item 
[^)  |  (  |  £j  Incomplete  item 

i  *  Q  Dictionary  inconsistent  with,  usage  elsewhere 


For  Kotational  Errors 

13.  Which  notation  type  was  in  error?  1 
/input/  (3  operator 

N  £3  //output//  £3  event 

(  )  I  O  $value$  £1]  simple  condition 

v~''/  *  litem!  £]]  compound  conditi 

I — |  *mode* 


14.  How  is  the  error  best  characterized 
|  |  incorrect  grouping 
□  incorrect  symbol 
n  a  incorrect  value 

ion 


Comments 

15.  Please  give  any  information  that  may  help  understand  the  change  and  its  cause. 


Fig.  1b  —  A-7  requirements  document  change  report  form  (reverse) 
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Changes  discussed  in  this  report  fall  into  one  of  two  categories:  error  corrections  and  non-error- 
corrections.  For  the  sake  of  brevity,  the  term  error  is  used  in  place  of  error  correction,  and  the  term 
modification  is  used  in  place  of  non-error-correction.  The  term  changes  is  used  to  refer  to  both  error 
corrections  and  non-error-corrections  where  no  distinction  between  the  two  need  be  made. 

The  data  distributions  presented  are  generally  displayed  in  accordance  with  the  categories  used  on 
the  CRF.  As  an  example,  error  distributions  use  the  categories  from  part  7  of  the  CRF:  "Clerical," 
"Ambiguity,"  "Omission,"  "Inconsistency,"  "Incorrect  fact,"  "Information  put  in  wrong  section,"  "Imple¬ 
mentation  fact  included,"  and  "Other." 

Before  proceeding  to  an  analysis  of  each  of  the  questions  previously  listed,  we  present  some  of 
the  general  characteristics  of  the  data  collected.  Figure  2  is  a  distribution  of  changes  by  type.  Of  the  88 
changes  reported,  79  were  errors.  Of  the  79  errors,  18  were  clerical.  Figure  3  is  a  distribution  of  errors 
by  type.  Figures  2  and  3  also  show  the  effort  involved  in  making  changes  and  in  correcting  errors.  Sec¬ 
tions  of  the  histograms  marked  T  (denoting  trivial)  indicate  changes  that  took  a  man-hour  or  less  of 
effort  to  make,  those  marked  E  (denoting  easy)  took  more  than  a  man-hour  but  no  more  than  a  man- 
day,  those  marked  M  (denoting  moderate)  took  more  than  a  man-day  but  no  more  than  a  man-week, 
and  those  marked  F  (denoting  formidable)  took  more  than  a  man-month.  There  were  no  changes  that 
took  more  than  a  man-week  but  no  more  than  a  man-month.  Only  one  formidable  error  has  yet  been 
found. 

Data  on  the  effort  required  to  understand  and  make  changes  are  provided  in  parts  5  and  6  of  the 
CRF.  These  data  are  the  basis  for  our  effort  estimates.  The  data  supplied  do  not  include  secretarial 
and  editing  effort,  but  only  that  effort  required  to  understand  why  a  change  has  to  be  made  and  what 
change  has  to  be  made  and  to  describe  the  change  in  form  sufficient  for  an  editor  or  typist  to  incor¬ 
porate  it  into  the  document.  In  addition,  nearly  all  changes  were  one-person  changes;  i.e..  one  person 
noticed  the  need  for  the  change,  did  the  research  necessary  to  understand  what  change  had  to  be  made, 
and  proposed  the  change.  Nearly  all  estimates  of  the  effort  to  make  changes  can  then  be  viewed  as  the 
effort  required  of  one  person.  Effort  estimates  given  in  this  report  are  obtained  by  assuming  that  trivial 
changes  took  one-half  hour  of  effort,  easy  changes  one-half  day,  and  moderate  changes  one-half  week. 


ERROR  OR  CORRECT  UNNECESSARY 
CORRECTION  A  PREVIOUS  INFORMATION 
CHANGE 


Fig.  2  —  Types  of  changes 
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CLERICAL  AMBIGUITY  OMISSION  INCONSIS-  INCORRECT  WRONG  OTHER 

TENCY  FACT  SECTION 


Fig  3  —  Types  of  errors 

An  estimate  of  six  man-weeks  for  the  one  formidable  error  was  obtained  through  discussions  with  the 
proposer  and  impiementer  of  the  change.  It  is  interesting  to  note  that  most  of  this  effort  (about  80%) 
involved  understanding  what  the  correction  should  be. 

The  effort  expended  in  producing  the  requirements  document  originally  was  17  man-months, 
including  both  development  and  review.  The  effort  expended  in  making  changes  was  about  1 1  man- 
weeks  and  the  effort  in  correcting  errors  was  about  10.5  man-weeks.  We  feel  these  are  small  in  com- 
parision  to  the  original  effort.  Discussions  with  those  who  wrote  and  those  who  changed  the  document 
revealed  that  many  of  the  people  who  were  making  changes  were  not  among  the  original  authors;  the 
effort  to  make  the  changes  consequently  contains  some  learning  effort  also. 

We  believe  that  one  reason  the  requirements  document  is  well  maintained  is  the  ease  of  making 
individual  changes  to  it.  As  will  be  shown  in  the  discussion  of  question  4,  typical  changes  take  about  2.4 
hours.  These  hours  are  often  expended  over  a  relatively  long  period  of  time,  since  the  need  for  a 
change  may  be  noted  without  specifying  the  change  to  be  made. 

The  list  of  questions  below  is  separated  into  three  categories:  those  questions  for  which  we  believe 
there  is  sufficient  data  to  discern  patterns  in  the  data  distributions  (questions  with  preliminary  answers), 
those  questions  for  which  there  is  insufficient  data,  and  those  questions  for  which  lack  of  data  may  be 
meaningful. 

Questions  with  Preliminary  Answers 

Are  I  he  External  Interfaces  Specified  C  'or  ret  tty? 

Eixtcrnal  interfaces  are  described  in  section  2  of  the  requirements.  To  answer  this  question  one 
must  find  all  errors  involving  section  2.  It  is  also  of  interest  to  segregate  these  errors  by  type  and  to 
estimate  the  effort  involved  in  correcting  them,  figure  4  shows  the  distribution  of  nonclerical 
external-interface  errors  by  type.  Clerical  errors  have  been  omitted  because  wc  assume  that  the  reason 
for  their  occurrence  is  unrelated  to  the  contents  of  the  section  of  the  document  in  which  they  occur. 
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ERROR  TYPE 

Fig.  4  —  Nonclerical  errors  in  section  2 


Section  2  of  the  requirements  is  of  particular  interest  because  it  contained  the  one  formidable 
error  found  so  far.  This  error  involved  the  incorrect  definition  of  a  coordinate  system  Most  of  the 
effort  in  correcting  it  was  consumed  by  a  study  of  the  use  of  coordinate  systems,  the  transformations 
between  them,  and  the  sensors  providing  navigational  information  for  the  aircraft.  The  effort  required 
to  correct  this  error  was  greater  than  the  effort  required  to  make  all  other  changes  to  the  document 
combined. 

We  estimate  the  effort  to  correct  the  nonclerical  errors  in  section  2  of  the  requirements  as  315 
man-hours  or  about  8  man-weeks.  This  was  far  more  effort  than  any  other  section  of  the  document 
and  about  75%  of  the  total  effort  to  correct  nonclerical  errors  so  far.  One  reason  for  this  may  be  that 
section  2  has  probably  received  more  use  as  a  design  specification  than  any  other  section  at  this  stage  of 
the  project;  consequently,  it  has  received  more  attention  than  any  other  section.  This  issue  will  not  be 
settled  until  the  project  has  ended. 

Is  the  Document  Easy  To  Change? 

Part  5  of  the  CRF  provides  an  estimate  of  the  effort  to  make  the  change.  Using  the  previously 
described  efTort  categories  and  estimation  algorithm,  based  on  the  responses  to  part  5  of  the  CRF  we 
can  estimate  the  effort  needed  to  make  changes  of  various  types  to  the  document.  The  total  effort 
required  for  all  changes  estimated  in  this  way  is  442  man-hours,  or  about  1 1  man-weeks  (note  that 
without  the  one  formidable-class  error,  the  effort  would  be  202  man-hours,  or  about  5  man-weeks). 
The  average  efTort  to  make  a  change  was  5  man-hours,  and  the  average  to  correct  an  error  of  any  type 
was  slightly  higher,  5.4  man-hours.  Without  the  formidable  error,  these  figures  are  sharply  reduced, 
becoming  2  3  and  2.4  man-hours  respectively. 

Although  there  are  few  data  on  modifications  (only  nine  have  yet  been  reported),  the  initial  indi¬ 
cation  is  that  they  require  less  effort  than  errors,  averaging  1.8  man-hours. 

/s  It  Clear  Where  a  Change  Has  To  Be  Made? 

Because  of  the  skewness  of  the  effort  distribution,  i.e.,  nearly  70%  of  the  changes  are  in  the  trivial 
category  as  shown  in  Fig.  5,  one  might  consider  the  "typical"  change  as  requiring  2.4  man-hours. 
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<  1  MH  1MH-1M0  I  MD  -1  MW  1  MW- 1  MM  >  I  MM 

EFFORT 


Fig.  5  —  EfVort  lo  change 

Analysis  of  the  data  from  part  3  of  the  CRF  shows  that,  for  all  but  one  change,  only  one  section  of  the 
document  had  to  be  examined  to  make  the  change.  We  can  now  characterize  the  typical  change  as  tak¬ 
ing  2.4  hours  and  only  requiring  inspection  of  one  section  of  the  document. 

Arc  Changes  Confined  to  a  Single  Section? 

Figure  6,  obtained  from  the  response  to  part  2  of  the  CRF,  shows  the  distribution  of  changes  by 
the  number  of  sections  of  the  document  changed.  Most  changes  only  required  modification  of  one  sec¬ 
tion  of  the  document.  Analysis  of  the  effort  for  single-section  changes  compared  to  multisection 
changes  shows  that  on  the  average  the  latter  required  about  27%  more  effort:  4.8  man-hours  for  single 
and  6.1  man-hours  for  multiple.  Not  only  does  it  seem  that  the  document  meets  the  goal  of  its  authors 
in  this  respect,  but  it  also  seems  that  this  was  a  worthwhile  goal. 
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Which  Sections  Have  the  Most  Errors'.’ 


Figure  7  shows  ihe  distribution  of  nonclerical  errors  by  section.  Sections  2  and  4  clearly  have  the 
majority  of  reported  errors.  This  is  likely  because  section  2  has  received  the  most  use  and  section  4 
second  most  at  this  stage  in  the  development  cycle.  Further  analy  sis  of  the  data  shows  that  the  distri¬ 
butions  of  error  types  in  these  two  sections  differ.  Figure  8  shows  these  distributions,  which  must  be 
considered  partial  as  yet. 

Which  Type  of  Table  lias  the  Most  Errors  '.’ 

Tables  tailored  to  the  A-7  flight  software  are  used  liberally  throughout  sections  2.  3.  4.  and  the 
dictionary.  The  four  principal  kinds  of  tables  used  are  mode-transition  tables,  event  tables,  selector 
tables,  and  condition  tables  (see  [14]  for  definitions  of  the  different  types  of  tables).  Of  the  61  noncler¬ 
ical  errors  so  far  discovered,  24,  or  39%,  were  errors  involving  tables.  More  than  half  of  these  (54  ) 
were  found  in  event  tables.  The  difference  in  error  incidences  may  possibly  be  attributed  to  the 
difference  in  nature  of  the  tables.  It  was  possible  to  do  consistency  and  completeness  checks  for  condi¬ 
tion.  selector,  and  mode-transition  tables  (a  procedure  for  validating  condition  tables  is  described  in 
(I3|),  but  not  forevent  tables. 
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The  distribution  of  nonclerical  table  errors  by  type  of  error  is  shown  in  Fig.  9.  This  distribution 
differs  markedly  from  the  corresponding  distribution  for  all  nonderical  errors  as  shown  in  Fig.  10. 
Omissions  dominate  the  table  errors,  whereas  incorrect  facts  dominate  the  distribution  of  all  nonderical 
errors.  Furthermore,  the  margin  of  domination  is  smaller  for  the  table  errors,  and  the  distribution  over 
omissions,  inconsistencies,  and  incorrect  facts  is  more  uniform  for  them.  There  are  several  possible 
explanations.  One  is  that  there  are  insufficient  data  yet  for  the  complete  pattern  to  appear.  A  second  is 
that  the  tables  may  be  just  a  good  way  of  organizing  information  so  as  to  make  completeness  checks 
easy. 


FACT 

ERROR  TYPE 


Fig.  9  —  Nonderical  table  errors 


FACT  SECTION 

ERROR  TYPE 

Fig.  10  —  Nonderical  errors  by  type 

Because  of  the  relatively  small  number  of  nonclerical  errors  involving  tables  so  far  found,  it  is 
premature  to  draw  firm  conclusions  concerning  the  usefulness  of  tables  in  general  or  event  tables 
specifically  from  the  data.  Now  that  patterns  concerning  table  errors  in  the  partial  data  have  been 
noticed,  we  will  continue  to  look  for  them  during  the  remainder  of  the  development  cycle. 

Patterns  of  the  sort  described  in  the  foregoing  provide  useful  feedback  to  the  developers. 
Unequal  error  distributions  may  mean  that  some  sections  of  the  document  have  not  been  as  carefully 
examined  or  as  fully  used  as  others,  and  require  further  review. 
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What  Use  of  the  Document  Reveals  the  Most  Errors'! 

Figure  1 1  shows  the  distribution  of  changes  according  to  the  way  the  document  was  being  used 
when  it  was  discovered  that  a  change  had  to  be  made.  The  distribution  is  derived  from  data  from  part 
1  of  the  CRF.  Since  the  project  is  currently  in  the  design  phase,  it  is  not  surprising  that  most  errors 
have  been  discovered  as  a  result  of  using  the  document  as  a  software  design  reference.  Recall  that  data 
collection  started  after  the  document  was  baselined  and  had  already  undergone  validation.  Clearly,  a 
number  of  errors  remained  even  after  the  initial  validation  process  was  completed.  Some  of  these,  but 
not  the  majority,  were  uncovered  by  later  validation  reviews.  (This  applies  especially  to  the  mode- 
transition  tables,  for  which  a  good  validation  algorithm  was  not  discovered  until  after  baselining.)  The 
initial  validation  process  included  reviews  both  by  the  authors  and  the  maintainers  of  the  current  A-7 
flight  software  at  NWC. 


reference 


USE  OF  DOCUMENT 
Fig  1 1  —  Discovery  of  need  for  change 


Are  Sections  3  (Modes)  and  4  (Functions)  Consistent  with  Each  Other ? 

Sections  3  and  4  of  the  requirements  document  are  complementary  views  of  the  system.  Section 
3  describes  the  set  of  modes  in  which  the  system  may  operate  and  the  events  that  cause  transitions 
between  modes;  it  may  be  viewed  as  a  state  description  of  the  system.  It  also  contains  information 
about  operations  to  be  performed  in  the  different  modes  and  data  items  that  are  used  and  that  may  be 
affected  when  the  system  is  in  the  mode.  Section  4  describes  the  functions  the  system  must  perform, 
in  which  modes  it  must  perform  them,  and,  for  each  function,  the  output  data  items  affected  by  the 
function. 

Because  sections  3  and  4  offer  complementary  views  of  the  system,  they  should  be  consistent  with 
each  other.  An  error  discovered  in  section  4  should  result  in  a  cross-check  to  section  3  for  a 
corresponding  error.  As  yet,  there  have  been  only  two  cases  where  corresponding  errors  have  been 
found  in  both  sections. 

Is  The  Dictionary  Complete,  Correct,  and  Consistent  with  the  Rest  of  the  Document,  and  H  ill  It  Remain  So? 

The  dictionary  serves  as  a  convenient  and  useful  means  for  abbreviating  and  cross-referencing  the 
requirements  document.  Terms  need  only  be  defined  in  one  place,  and  those  unfamiliar  with  the 
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meaning  of  a  particular  term  can  quickly  find  its  definition.  The  dictionary  also  serves  as  an  important 
tool  for  abstraction.  The  definition  for  a  term  such  as  slant  range  may  be  used  without  the  need  to 
know  how  to  calculate  it  or  what  data  are  needed  to  calculate  it. 

Figure  12  shows  that  most  dictionary  changes  involved  the  dictionary  and  at  least  one  other  sec¬ 
tion.  In  every  case  but  one,  these  changes  were  error  corrections  that  included  adding  a  new  term  to 
the  dictionary.  One  may  think  of  these  changes  as  adding  abstraction  to  the  document.  The  one  excep¬ 
tion  was  a  definition  that  was  incomplete. 


CONTENTS  0  2  3  A  9  II  INDEX 

DOCUMENT  SECTION 

Fig  12  —  Changes  by  section 


Changes  that  were  confined  to  the  dictionary  alone  were  usually  incorrect  definitions;  i.e..  the 
term  defined  corresponded  to  some  "ideal"  quantity  whose  accepted  definition  did  not  ouite  correspond 
to  the  requirements  dictionary  definition,  in  only  one  case  was  the  dictionary  found  to  be  inconsistent 
with  usage  elsewhere  in  the  document. 

For  its  first  14  months  of  use,  the  dictionary  appears  to  have  been  well  maintained.  Changes  else¬ 
where  in  the  document  stimulated  the  appropriate  changes  in  the  dictionary.  There  seem  to  be  few 
inconsistencies  with  the  rest  of  the  document. 

Questions  for  Which  Lack  of  Data  May  Be  Meaningful 

Is  Externally  Visible  Behavior  Only  Specified  Without  Implying  a 
Particular  Implementation  ? 

Data  to  answer  this  question  arc  supplied  by  part  7  of  the  CRF.  Including  an  implementation  fact 
in  the  requirements  is  considered  an  error.  No  errors  of  this  type  have  yet  been  reported. 

Are  the  Appropriate  External  Interfaces  Specified '! 

Data  to  answer  this  question  arc  supplied  by  parts  2,  3,  and  7  of  the  CRF.  Currently,  no  change 
has  involved  adding  a  new  external  interface  or  deleting  an  existing  interface  from  the  document. 


14 


NR!  K!  I’OK  I  X44^ 


Does  the  Document  Contain  Unnecessary  Information  ' 

No  changes  involving  removal  of  unnecessary  information  (see  part  4  of  the  CRH  have  yet  been 
made,  nor  have  any  errors  involved  the  inclusion  of  an  implementation  fact. 


Questions  Not  Currently  Answerable 

There  are  insufficient  data  available  to  answer  the  following  questions. 

Are  the  Changes  Likely  to  Occur  Predicted  Correctly  ? 

Requirements  for  the  NRL  version  of  the  A-7  flight  program  were  frozen  at  the  start  of  the 
redevelopment  project.  Consequently,  data  to  answer  this  question  will  not  become  available  until  the 
NRL  flight  program  is  completed  and  changes  to  the  program  are  then  considered. 

Is  the  Proper  Set  of  Undesired  Events  Described ? 

Is  the  Notation  Used  Unambiguous ? 

Where  Do  the  Most  Changes  Have  To  Be  Made'.' 

Which  Subsections  of  Sections  2  (Data  hems).  J  (Modes),  and  4 
(functions)  Are  Most  Error-Prone? 

CONCLUSIONS 

We  have  two  main  objectives  in  monitoring  the  changes  made  to  the  A-7  software  requirements 
document.  One  objective  is  to  investigate  the  feasibility  of  applying  goal-directed  data  collection  con¬ 
currently  with  document  maintenance.  Similar  techniques  have  been  successfully  applied  to  code  dur¬ 
ing  program  development  [17],  A  second  objective  is  to  try  to  measure  the  success  with  which  the  A-7 
requirements  authors  met  their  objectives.  We  believe  the  latter  objective  to  be  particularly  important 
because  the  A-7  redevelopers  are  attempting  to  use  a  methodology  to  produce  an  engineering  model.  If 
they  succeed,  it  is  important  to  know  the  weak  and  strong  points  of  the  model.  If  they  fail,  it  is  impor¬ 
tant  to  know  what  the  troublesome  areas  are  both  in  the  application  of  particular  techniques  and  in  the 
integration  of  different  techniques. 

Two  kinds  of  conclusions  may  be  drawn  from  this  study;  one  kind  concerns  the  data-collection 
method  itself,  and  one  kind  concerns  the  A-7  software  requirements  document.  Conclusions  concern¬ 
ing  the  data  collection  method  are: 

•  The  data  collection  method  seems  to  be  feasible  and  useful.  By  integrating  it  with  the 
configuration  control  process  we  have  tried  to  keep  down  the  overhead  associated  with  it. 
Data  distributions  to  answer  questions  of  interest  both  to  the  A-7  redevelopers  and  to 
software  engineers  are  producible. 

•  Data  distributions  based  on  partial  data  seem  to  provide  some  useful  feedback  to  the 
redevelopers.  As  an  example,  error  distributions  that  show  uneven  patterns  of  error  detec¬ 
tion  may  indicate  that  some  document  sections  need  further  attention. 

•  As  patterns  are  discerned  in  the  data,  new  questions  of  interest  emerge.  As  an  example, 
comparison  of  error  distributions  across  different  sections  of  the  document  shows  that  the 
distributions  often  differ  significantly.  There  is  no  obvious  explanation  for  these  differences. 
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but  many  hypotheses  can  be  formed  to  explain  them.  We  expect  that  answers  to  some  of  the 
newly  formed  questions  will  be  available  later  in  the  project;  others  will  probably  not  be 
answerable  with  the  data  currently  being  collected. 

Conclusions  concerning  the  requirements  document  are  generally  answers  to  the  questions  dis¬ 
cussed  in  the  previous  section.  Some  of  the  more  significant  conclusions  are: 

•  The  document  seems  to  be  easily  maintained.  The  low  effort  to  correct  a  "typical"  error  sup¬ 
ports  this  conclusion.  It  is  important  to  note  that  all  the  requirements  errors  must  be  found 
in  order  to  produce  a  correct  system  whether  or  not  the  requirements  document  is  updated  to 
reflect  the  corrections.  As  a  result,  the  effort  involved  in  understanding  requirements  errors 
comes  free  (from  the  viewpoint  of  updating  the  requirements  document). 

•  The  document  is  worth  maintaining  The  uses  to  which  it  is  being  put,  as  taken  from  par;  I 
of  the  CRF,  show  that  it  is  being  heavily  used  during  design  and  is  being  used  for  mainte¬ 
nance  of  the  existing  A-7  flight  software. 

•  Despite  a  validation  process  that  included  both  the  authors  of  the  document  and  the  main¬ 
tained  of  the  existing  flight  software,  a  number  of  errors  remained  in  the  document  after  it 
was  validated  and  baselined.  The  uneven  distribution  of  errors  by  sections  suggests  that  a 
significant  number  of  errors  may  remain  in  the  sections  that  have  been  lightly  used. 

•  The  document  seems  to  be  well  structured  in  that  changes  can  be  made  in  one  section 
without  requiring  many  changes  elsewhere. 

We  expect  to  continue  data  collection  through  the  entire  A-7  flight  software  redevelopment  pro¬ 
ject.  Data  collection  will  be  tailored  to  the  project  phase  and  the  techniques  being  used  We  have 
presented  here  a  description  of  the  data  collection  techniques  and  results  from  analysis  of  partial  data 
because  we  would  like  to  encourage  others  to  pursue  similar  projects. 
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